In terms of network security, oftentimes a network is under attack for those intent on hacking into a computer system to either disrupt the system or to take over control of the computers residing on the system. Networks employ spam filters and Malware detection as well as detecting login attacks. Most of these approaches involve either signature-based approaches or anomaly-based detection to see if a computer system is under attack.
The problem with prior approaches to network security is that while they are relatively robust in detecting massive attacks that occur for instance within milliseconds, they are unable to detect attacks which are sporadic or take place over long periods of time. In some attacks, the attacker tries to, access system passwords through trial and error. However, if passwords are tried over days, weeks and months, as opposed to seconds, then very cumbersome manual approaches to detecting the attack are sometimes tried. The result however, is that such attacks are not detected at all or are detected too late.
Moreover, while there are a number of reasons to detect different types of attacks, there are nonetheless large numbers of false alarms. These can be due to minor spam attacks or system noise and it is only with difficulty that one can isolate a major high level attack from mere annoyances.
While math models to represent potential behaviors in a network such as artificial intelligence, neural networks and the like use statistical models to ascertain behavior, these models are insufficient for the more stealthy approach of attacking a system periodically over a long period of time.
Thus, there are detectable events when attackers seek to scan a network that because of their complexity are difficult to detect either by a manual inspection of logs or by using models of potential behaviors. Thus, complex attack models involving sophisticated patterns, frequencies or repetitions have heretofore not been automatically detectable, or are masked by false alarms.
As will be appreciated, attackers oftentimes try to determine if they can recognize a system and characterize it by detecting the type of operating system that is utilized and by detecting the type of applications running on the system. This type of scan is called a TCP scan which is an acronym for the Transmission Control Protocol. The TCP protocol is a standard protocol involving a three-way handshake in which a client tries to connect to a server, the server acknowledges the attempt and then the client responds with a final thank you in a three-way handshake. If the scan involves an incomplete transaction, the attacker will try to connect to a server and the server will respond. However the client will never acknowledge the servers response, thereby leaving the server hanging and open. This is what is referred to as an incomplete TCP connection and is one mechanism for denying service.
Moreover, not only are single incomplete sessions an indication of an attack, when there are a large number of incomplete sessions, that also is a symptomatic of a scan.
Note, not only are passwords attempted in order to enter into a system, network sensors can include packet and protocol content inspections. Content inspections look at various parameters such as an http web page and the actual words on a page.
Malware relates to looking for known signatures or known content and behavior models may be employed to ascertain when a system is connecting to a large number of internal hosts. Typically if the system reaches out to a web server, the server may be instructed for instance to double click on a page and go to another site for more information. If for instance the network experiences ten connections per second after one connects to a page for a couple of seconds, this may indicate a Malware attack.
As to network flow detection, flow rates which involve connections to different hosts, can if certain thresholds are exceeded indicate a scan.
More particularly, if an entity seeks to scan a system utilizing an attack mechanism, for instance three times in a row within a short time period, then for simple attacks one can quickly recognize the short-duration repetitive behavior. Such attacks such as SSH login attacks which occur as many as a thousand times a second are readily detectable.
However, if the attack occurs for instance a couple of times throughout a day or perhaps a few times in the succeeding days over extended periods of time, it is very difficult to recognize such attacks with present methods.
Those systems which react to the sensing of an attack so as to report it every time it occurs are plagued with false alarms, called noise. This means that attacks spanning long periods of time are not very easy to pick off. Thus, if an entity seeks to break into a computer with an automated password attack by trying a large number of passwords; if the attack occurs with a massive number of passwords per second, it is quite easy to detect such an attack.
However, if the attacker tries the various passwords over an extended period of time, then it is very likely that such attacks will go on unnoticed.
So-called nation state attacks which seek to gain control over sensitive computers needs only to have one correct password recognized in order to be able to control the computer. Thus, if the attacks occur for instance over a number of months, the instant that a password is successful, the computer is compromised.
Those events which indicate that a scan or attack is in process include incomplete TCP handshaking in which there are no answer-backs. Also the number of TCP resources invoked, if large, is also an indication of the presence of an attack. Moreover, one can analyze packet and protocol contents to sense an attack, as well as utilize conventional Malware detection systems and/or network flow detection.
As noted above, if there are certain Malware attacks, one seeks to identify these attacks due to the particular recognized signature of the attack. These signatures can include the transmission of malicious code and one may not be sure utilizing conventional Malware detection whether a virus or Trojan attack is in fact taking place.
If one looks at the suspected malicious code over a long time period, one may for instance see five occurrences in ten minutes. It is noted that if one sees five occurrences in ten minutes, one is potentially seeing half an occurrence per minute and one might decide that half an occurrence per minute indicate the presence of a Malware attack.
Thus, two problems occur in intrusion detection systems. The first is a false firing on a signature, noting that the false firing may occur hundreds to thousands of times a second, accounting for millions of events during a day, all of which need to be analyzed. Often such attacks involve low quality events that can be ignored.
Without frequency or recurrence filtering, it is very difficult to arrive at a model of behavior characterizing attacks that does not result in an abundance of false-positives.
While password attacks historically utilize a brute force approach. A brute force attack is very clear and obvious when one gets thousands of bad password attacks on an account. However, it is extremely difficult to ascertain that attacks have occurred if for instance the attacker tries one password a minute, or one password a day which tends to fall below the firewall rules that are for instance set to see if three failed password attempts have been made.
Due to the sophistication of attackers, most attackers now extend their attacks over much longer durations such as days, weeks or months and even years, and it is now a necessity to be able to track such attacks with sophisticated analysis.
It is possible to take a look at 10 g histories in order to derive the information necessary to ascertain that an attack has occurred or is process. However, it is clear that such manual approaches or even simple automatic approaches are incapable of countering such long term behavior.
In summary, artificial intelligence and neural networks when properly trained still cannot indicate attacks that occur over very long periods of time. Moreover, simple approaches do not work well and intusion detection systems will have to require a very high level of computer automation in order to detect the various patterns that an attacker might utilize. Thus, there is a requirement for a threat detection system that depends on a sequence of many events happening over a long period of time to achieve a low false alarm rate.
Note for purposes of the following discussion the following definitions apply:
anomalous packet=a network packet that is in some way unusual compared to other packets.
unknown network flows=a network communication which cannot be deciphered, or categorized as a known communication protocol
shellcode detection=‘shellcode’ refers to a malicious piece of code used in part or whole to compromise or takeover a computer system.
DOS response=DOS is a Denial of Service attack, as when an attacker sends a lot of connection requests to a web server, hoping to overload the server. A DOS response is something you would do to mitigate the denial of service, for instance block the attackers communications at a firewall.
TCP flows=a TCP flow is a network communication using the TCP network protocol, also known as a TCP session.
Host=IP, IP is short for an internet protocol address. A Host is a desktop, or a server, each are hosts.
Port=a connection node.
IP flow=IP to IP communication, one computer talking to another in any manner, shape or form.
TCP Flow=Two hosts communicating using the TCP protocol. Tracking this requires a Client IP (the initiator), and the Server IP and Server Port. When one goes to google.com this is over a TCP flow.
UDP flow=IP:Port to IP:Port
TCP Server=a Server IP and the Servers Port Track options include all, host, port, IP flow, TCP flow, UDP flow and TCP server. These are specific options a user would enable when running a Network Intrusion Detection Defense and Response System (NIDAR). Enabling these options tells the NIDAR to track these values across a series of network sessions (TCP flows).